I think "R&D" is a misleading category - it comprises a LOT of activities with different uncertainty, type, scope, and timeframe of impact. For tax and reporting purposes, a whole lot of not-very-research-ey software and other engineering is classified as "R&D", though it's more reasonably thought of as "implementation and construction".
Nordquist's "Innovation" measure is very different from economic reporting of R&D spending. This makes the denominator very questionable in your thesis.
Perhaps more important, returns are NOT uniformly distributed. Even successful "pure research" projects have a MIX of short/medium-term localized benefits and longer/broader impacts, and research insitutions are (mostly) pretty good at managing BOTH grants and licensing/product development as funding and "value capture" mechanisms.
William Nordhaus estimates that firms recover maybe 2% of the value they create by developing new technologies.
Isn’t this the wrong metric? 2% of the value of a new technology might be a lot of money, far in excess of the R&D cost required to create it.
Some observations from startup culture about under investment in r&d even in one of the world's supposed hot beds of innovation:
1: investors want a return soon and often inadvertantly push for lower variance strategies.
2: it is difficult to sell people on entirely new product categories.
3: people give poor feedback on new product categories and so it is up to the taste of the inventor to refine the product development.
4: legal structures are set up for benefit of those good at navigating legal structures, see why most politicians are lawyers. Many inventors wind up forced out of returns to their inventions.
5: manufacturing new product categories often requires modifications to existing manufacturing capacity, which hurts economies of scale. Often it is difficult to find anyone willing to help manufacturer your thing at all.
6: inventors are cognitively weird and thus socially weird, and fail many tacit social handshakes in the course of trying to coordinate with others.
7: innovators who are both weird and have the requisite taste have a hard time getting along with each other since they have strong opinions, including some strong wrong opinions.
8: there is competition between time spent innovating and time spent networking into groups that would support the business considerations.
9: selection of good business partners is a separate skill from innovating.
10: incumbents will use their existing and considerable cash flows to push back against new entrants, sometimes illegally.
11: consumers on average have short time horizons, so innovations that drive bigger improvements over time can lose out to the fast easy fix that kicks cans down the road.
12: if an innovation is both cheap and something a consumer only needs to buy once it can be hard to support the business.
13: established product areas can have a minefield of IP such that the obvious combinations of features would be legally prohibited from being sold without unavailable license deals.
14: an innovation that interacts with a politicized aspect of the economy, eg recycling, will face strong pushes from unexpected directions unrelated to what the product tries to physically do.
15: an innovation that mainly helps people with problems that involve social desirability bias will be hard to market.
16: innovations that put people out of a job have a built in counter lobbying force.
13: established product areas can have a minefield of IP such that the obvious combinations of features would be legally prohibited from being sold without unavailable license deals.
Do you have links to documented cases where products seem to be unavailable for these reasons?
specifically documented? No. I think some of the obvious examples are in things like batteries, materials science, and computer parts, where there are strong IP fights. Eg AMD, Intel, Nvidia, and ARM all license some but not all of their core tech to their rivals. I'm actually pretty confused about how they determine when to do this vs not, but would guess that this is at least somewhat inefficient by over optimizing on short term gains vs the more nebulous future payoffs of what would be enabled with more licenses.
Legal fight are the result of products that the market wants being made available even when it violates license rights. Companies then just pay the what the court tells them is a reasonable price for the patent violation.
It might be that in practice the friction makes certain products not available instead of just increasing their prices a bit, but I would want to hear from an industry insider that this is a significant problem to believe that.
They model the returns to R&D spending like this: imagine stopping all R&D spending for a single year. You’d save several hundred billion dollars upfront, but there would be no economic growth,1 so we’d miss out on a ~2% increase in per capita GDP.
It's a cute argument and I like it, though I note that probably-most economic innovation does not come from things-called-R&D-spending. For instance, I'd expect selection pressure on businesses is a major mechanism. People try things and see what makes money, and that's not usually classified as "R&D".
When the government wants electric car production it has a choice to invest directly into electric car R&D or to subventionize electric cars. The United States spends much more money on subventions than on R&D.
I would guess that if you ask the lawmakers they would say something like "Electric car companies are much better at investing money into developing better cars than the government happens to be, so the subventions are the more effective policy tool than paying the same amount of money for basic research".
I don't think this is a pure matter of incentives but simply of the ideology of privatization. It might be wrong and in reality direct government R&D is more effective than subventions but that's an argument you have to make more directly.
The owner of a firm investing in R&D doesn’t account for all the benefits their technology might bring to non-paying consumers and firms, but they do care about the benefits that R&D will bring to the firm long into the future, even after their death. One part of this is that owners don’t face term limits that incentivize pump-and-dump attempts to garner voter support.
This does not match my expectations, even if it agrees with how I would feel were I the owner.
For example, the top ten Nasdaq companies spent ~$222B between them on R&D, which is almost half of the $463B spend from the private sector.
But I notice these are publicly traded companies, where the owner is in fact the shareholders. The average holding period for stocks is less than a year; if memory serves in 2020 it was less than 6 months. The average shareholder definitely does not care about the value of R&D to the firm long after their deaths, or I suspect any time at all after they sell the stock.
I notice that Amazon and Tesla are both on the list, and maybe by owner we really mean founder; I could easily see Bezos and Musk feeling that way about R&D at their companies. But most of R&D spend from the private sector is not from founder-run companies; it is from professionally managed companies with hired CEOs and executives.
On that note, I feel like I have seen plenty of cases where executives and CEOs do precisely the pump-and-dump phenomenon on the project side. Maybe executives have longer tenures than legislators, so it makes more sense for them to take a longer view? Yet when I check, CEOs have on average shorter tenure than Congress does: the CEO has 7.2 years; Representatives and Senators have 8.9 and 11 years respectively. I also note that the CEO has much more power over the budget than legislators do; the legislator at least requires the agreement or indifference of a majority of the legislature to get R&D stuff added, whereas the CEO can very often decide budget priorities unilaterally.
That being said, this doesn't seem to change the overall thrust of the post. My suspicion lands on the 2% captured value number as being misleading; for the individual company, 2% of a huge number can easily be more than 98% of a much smaller number. I'm also curious about how that number 2% is built, so my next step here is to check out the Nordhaus paper more deeply.
"The average shareholder definitely does not care about the value of R&D to the firm long after their deaths, or I suspect any time at all after they sell the stock."
This was addressed in the post: the price of the stock today (when its being sold) is a prediction of its future value. Even if you only care about the price that you can sell it at today, that means that you care about at least the things that can lead to predictably greater value in the future, including R&D, because the person you're selling to cares about those things.
Also worth noting: the reason that the 2% value is meaningful is that if firms captured 100% of the value, they would be incentivized to increase the amount produced such that the amount they create would be maximally efficient. When they only capture 2% of the value, they are no longer incentivized to create the maximally efficient amount (stop producing it when cost to produce = value produced). This is basically why externalities lead to market inefficiencies. The issue isn't that they won't produce it at all, it's that they will underproduce it.
I wonder if the point about relative investment also holds if one restricts one's perspective to technologies needed to ensure security, i.e. military gear and similar technologies.
1. Categories
Fundamental Research = State
Applied Research = Companies
.. is a common paradigm, and - while grossly too simplified - makes some sense: the latter category has more tangible outputs, shorter payback etc. In line with @Dagon's comment, at the very least these two broad categories would have to be split for a serious discussion of whether 'too much' or 'too little' is done by gvmt and/or companies.
2. Incentives!
I've worked in a research startup and saw the same dollar go much further in producing high-quality research outputs than what I've directly experienced in some places (and from many more places observed) in the exact same domain in state-sponsored research (academia) where there is often a type Resource Curse dynamics.
My impression is, these observations generalize rather well (I'm talking about a general tendency; needless to say, there are many counterexamples; often those wanting to do serious research are exactly attracted by public research opportunities, where they can do great work). Your explanation leaves out this factor; it might explain a significant part of the reluctance of the state to spend more on R&D.
This does not mean the state should not do more (or support more) R&D, but I think there are very important complexities the post leaves out, limiting the explanation power.
One reason that economically rational firms might produce "too much" R&D for their shareholders: competition! In many industries, if you stop innovating your company will fall behind quickly, and your customers will go elsewhere.
Happily this Red Queen's Race has large positive externalities, and so coordinating to reduce such investments is generally illegal ('restraint of trade').
Discovering new technologies is the only way to get long-term economic growth. Rote expansions of existing technologies and machines inevitably hit a ceiling: replacing and repairing the existing infrastructure and capital becomes so expensive that there is no income left over for building extra copies. The only way out of this is to come up with new technologies which create more income for the same investment, thus restarting the feedback loop between income growth and investment.
So R&D is extremely valuable. But most of the gains from R&D accrue to external parties. William Nordhaus estimates that firms recover maybe 2% of the value they create by developing new technologies. The rest of the value goes to other firms who copy their ideas and customers who get new products at lower prices. Firms don’t care much about the benefits that accrue to others so they invest much less in R&D than the rest of us would like them to.
Governments, on the other hand, collect much more of the benefits from new technologies. They get to tax the entire economy so when benefits spillover across firms and consumers, they still come out ahead. They don’t collect on international spillovers but for large economies at the frontier of technological growth, like the US, they internalize a large chunk of the value from R&D, much more than 2%.
All of this is a setup for a classic externalities problem. There’s some big benefit to society that private decision makers don’t internalize, so we should rely on governments to subsidize R&D closer to its socially optimal level.
But in fact, the private sector spends ~4x more than the public sector on R&D: $463 vs $138 billion dollars a year. One explanation for this might be that the extra $138 billion is all that was needed to bump up private spending to the social optimum, but this doesn’t seem to hold up in the data. One piece of evidence that we are still far off the socially optimal spending on R&D comes form a simple accounting of the average returns to R&D by Larry Summers and Benjamin Jones.
They model the returns to R&D spending like this: imagine stopping all R&D spending for a single year. You’d save several hundred billion dollars upfront, but there would be no economic growth,1 so we’d miss out on a ~2% increase in per capita GDP. The upfront savings only happen once, but next year when we start R&D up again we have 2% less to invest so we grow less, and next year we’re still behind, and so on ad infinitum.
These repeated long term costs of missing R&D add up to outweigh the upfront benefits from the money we’d save under most reasonable views of the value of economic growth and the contribution of R&D to that growth. Summers and Jones suggest that each dollar spent on R&D creates $14 dollars of value on average!
So this leaves us with a question: Why aren’t governments taking this free lunch? Why are they letting the weakly incentivized private firms outspend them on the world’s most important positive externality?
This puzzle is explained by the distinction between spatial and temporal externalities. The argument we made above about how the government collects on spillovers between firms and customers because it taxes the entire economy is true, but only if those spillovers happen fast. No decision maker in government today benefits from R&D spillovers that accrue 20 years later. In fact, they are often made worse off since R&D spending has immediate costs and only future benefits. Perhaps governments as a single abstract entity internalize the country wide benefits of R&D that accrue decades in the future, but no actual decision maker working in government stands to gain.
Market actors, on the other hand, are better incentivized to care about temporal externalities. The owner of a firm investing in R&D doesn’t account for all the benefits their technology might bring to non-paying consumers and firms, but they do care about the benefits that R&D will bring to the firm long into the future, even after their death. One part of this is that owners don’t face term limits that incentivize pump-and-dump attempts to garner voter support. But even if the owner of a company knows they are retiring soon, they still have good reason to care for the long term value of their firm. This is because when they go to retire and sell the company, they are paid the present discounted value, which takes into account the company’s future prospects. In many industries R&D is a major determinant of these future prospects.
There is more going on in government’s decision of how much R&D to fund than their greater care for spatial externalities over temporal ones. This story doesn’t explain why they spend $50 billion on long-term basic research without short term benefit, for example, but it does explain why private firms are doing more to provide for this positive externality than governments are.
1 The qualitative conclusion that R&D spending has large average returns holds under significant relaxations of this assumption.